Optical biometric imaging device and method of operating an optical biometric imaging device

ABSTRACT

Method for determining image reconstruction parameters in an optical biometric imaging device comprising a plurality of microlenses forming a microlens array and an image sensor arranged to receive light having passed through the microlenses, the method comprising: by the image sensor, capturing a plurality of sub-images together representing an image of a biometric object in contact with a sensing surface of the biometric imaging device, each sub-image corresponding to a respective microlens; and determining a demagnification factor based on at least a subset of the plurality of sub-images.

FIELD OF THE INVENTION

The present invention relates to a method for determining imagereconstruction parameters in an optical biometric imaging device.

BACKGROUND OF THE INVENTION

Biometric systems are widely used as means for increasing theconvenience and security of personal electronic devices, such as mobilephones etc. Fingerprint sensing systems, in particular, are now includedin a large proportion of all newly released consumer electronic devices,such as mobile phones.

Optical fingerprint sensors have been known for some time and may be afeasible alternative to e.g. capacitive fingerprint sensors in certainapplications. Optical fingerprint sensors may for example be based onthe pinhole imaging principle and/or may employ micro-channels, i.e.collimators or microlenses to focus incoming light onto an image sensor.

There is also a desire to integrate fingerprint sensors in the displaypanel of a user device such as a smartphone in order to achieve“in-display” fingerprint sensing over a larger part of the display area.Optical fingerprint sensors in particular have shown promise for displayintegration where an optical fingerprint sensor can be arrangedunderneath an least partially transparent display panel.

For optical fingerprint imaging, the distance between the lenses of thesensor and the object to be imaged, i.e. the object distance, influencesthe image properties and an optical sensor it typically calibrated tooperated properly for a certain object distance. The calibration of theoptical sensor is often performed during production using a calibrationtarget such as a bar target.

It may also be desirable to calibrate the optical sensor once the deviceis in use to account for changes in the object distance which may arisefrom wear and tear of the device or from structural changes of thedevice in which the optical sensor is arranged. One situation inparticular which may have a large influence on eh image properties of anoptical sensor in a user device is if a screen protector is arrangedover a display, in which case the object distance is changedsignificantly. Such a change may lead to a loss of resolution in aresulting image which is not possible to correct for. It would thus bedesirable to perform a new calibration once a screen protector has beenarranged on a display.

However, calibration using a dedicated calibration target is complicatedand therefore not practical once a device has left the productionfacility and is in use. Moreover, it is not necessarily known when a newcalibration is needed.

In view of the above, it is desirable to provide an improved approachfor calibrating an optical biometric sensor.

SUMMARY

In view of above-mentioned and other drawbacks of the state of the art,it is an object of the present invention to provide a method fordetermining image reconstruction parameters in a biometric imagingsensor without the use of a specific calibration target.

According to a first aspect of the invention, there is provided a methodfor determining image reconstruction parameters in an optical biometricimaging device comprising a microlens array and an image sensor arrangedto receive light having passed through the microlens array, the methodcomprising: by the image sensor, capturing a plurality of sub-imagestogether representing an image of a biometric object in contact with asensing surface of the biometric imaging device, each sub-imagecorresponding to a respective microlens, wherein sub-imagescorresponding to adjacent microlenses are partially overlapping; anddetermining a demagnification factor based on at least an overlap asubset of the plurality of sub-images.

In a biometric imaging device based on microlenses, the image sensorwill capture a plurality of sub-images corresponding to the plurality ofmicrolenses. When processing the captured images in order to form acomposite image to be used for verification and/or authentication, thedemagnification factor is used to describe the optical properties of theimaging device. The demagnification factor can be defined as a ratiobetween an object width and an image width where the object width is thewidth of an object located in the object plane and the image width isthe width of that object in the image plane. The demagnification factoris thereby used as an important image reconstruction parameter forforming the composite image.

The present invention is based on the realization that a demagnificationcan be determined from the captured sub-images with sufficient accuracyfor using in image reconstruction without performing calibration. Inparticular, the demagnification factor can be determined based on thepartial overlap of adjacent sub-images. Accordingly, a change in objectdistance would be reflected by a corresponding change overlap betweenadjacent sub-images, which can be used to determine the demagnificationfactor where the updated demagnification factor can be used in imagereconstruction. Thereby, calibration of the imaging device can beperformed during normal image capture without the need for any specificcalibration target.

An advantage of the described invention is that appropriate imagereconstruction parameters can be derived when the physical distancebetween the microlenses and the object to be image has changed, and thataccurate biometric imaging can be performed once the updated imagereconstruction parameters are determined.

According to one embodiment of the invention, the method comprises:determining a spatial offset between two captured adjacent sub-imagescorresponding to two adjacent microlenses; and determining thedemagnification factor based on the determined spatial offset.

According to one embodiment of the invention, determining a spatialoffset comprises determining a cross correlation between the twocaptured adjacent sub-images.

According to one embodiment of the invention the method furthercomprises: selecting a subset of microlenses having the samedemagnification factor; determining an X-dimension and an Y-dimensioncross correlation vector for each pair of microlenses in the subset ofmicrolenses; forming an average X-dimension and Y-dimension crosscorrelation vector from the determined cross correlation vectors;summing the X-dimension and Y-dimension cross correlation vector; anddetermining a spatial offset based on the summed cross correlationvector. In principle, the cross correlation vector may be defined in anarbitrary direction. However, it may be computationally advantageous touse the X- and Y-directions defined by pixels in the sub-image.

According to one embodiment of the invention, the method furthercomprises determining the demagnification factor based on the spatialoffset and a known pitch between adjacent microlenses. Thereby, thedemagnification factor can be determined independently of other imagereconstruction parameters and without prior knowledge of thedemagnification factor.

According to one embodiment of the invention, the method may compriseperforming stitching to form a full image from a plurality of sub-imagesbased on a predetermined demagnification factor and a known microlenspitch; determine a figure of merit for the full image based on apredetermined set of image properties; comparing the figure of meritwith a predetermined figure of merit threshold value; and if the figureof merit is below the predetermined figure of merit threshold value,change the demagnification factor and determine a new figure of merit.The method thus uses a figure of merit using a known demagnificationfactor for a known object distance. This in turn requires that ademagnification factor is known, for example from a calibrationperformed during production.

According to one embodiment of the invention, the predetermined set ofimage properties is at least one of image contrast and frequencycontent. Accordingly, parameters which may objectively describe thequality of the image are used to determine the figure of merit. Thefigure of merit may be a composite based on different image propertiesand it is also possible to use different figures of merit for differentimage properties.

According to one embodiment of the invention the predetermineddemagnification factor may be acquired using a calibration image target.

According to one embodiment of the invention, the method furthercomprises optimizing the demagnification factor by iteratively changingthe demagnification and the quality radius to determine a maximum figureof merit.

According to one embodiment of the invention, the method may furthercomprise using the determined or optimized demagnification factor in animage reconstruction process to form an image to be used for biometricverification.

According to a second aspect of the invention, there is provided abiometric imaging device comprising: an image sensor; and a microlensarray arranged to redirect light from a sensing surface of the imagingdevice towards the image sensor, wherein the image sensor is configuredto capture a plurality of sub-images together representing an image of abiometric object in contact with the sensing surface, wherein sub-imagescorresponding to adjacent microlenses are partially overlapping, eachsub-image corresponding to a respective microlens, and to determine ademagnification factor based on an overlap of at least a subset of theplurality of sub-images.

According to one embodiment of the invention, the imaging device isfurther configured to determine a spatial offset between two capturedadjacent sub-images corresponding to two adjacent microlenses anddetermine the demagnification factor based on the determined spatialoffset. Moreover, the microlenses may for example be arranged in theform of a hexagonal array.

Additional effects and features of the second aspect of the inventionare largely analogous to those described above in connection with thefirst aspect of the invention.

Further features of, and advantages with, the present invention willbecome apparent when studying the appended claims and the followingdescription. The skilled person realize that different features of thepresent invention may be combined to create embodiments other than thosedescribed in the following, without departing from the scope of thepresent invention.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other aspects of the present invention will now be describedin more detail, with reference to the appended drawings showing anexample embodiment of the invention, wherein:

FIG. 1 schematically illustrates a biometric imaging device according toan embodiment of the invention;

FIG. 2 is a method outlining the general steps of a method according toan embodiment of the invention;

FIG. 3 is a flowchart outlining steps of a method according to variousembodiments of the invention;

FIG. 4 is a flowchart outlining steps of a method according to variousembodiments of the invention;

FIG. 5 is a flowchart outlining steps of a method according to variousembodiments of the invention; and

FIG. 6 schematically illustrates a smartphone comprising a biometricimaging device according to an embodiment of the invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

In the present detailed description, various embodiments of thebiometric imaging device and method for determining image reconstructionparameters according to the present invention are mainly described withreference to an optical fingerprint sensing device arranged under adisplay panel of an electronic device such as a smartphone. However, theimaging device may also be used to capture other biometric features,such as palmprints, and the imaging device may be integrated in a widerange of applications.

FIG. 1 schematically illustrates a portion of a biometric imaging device100 according to an embodiment of the invention. The biometric imagingdevice 100 is here arranged under an at least partially transparentdisplay panel 102. However, the biometric imaging device 100 may bearranged under any cover structure which is sufficiently transparent, aslong as the image sensor 108 receives a sufficient amount of light tocapture an image of a biometric object in contact with the outer surfaceof the cover structure 102, such as a fingerprint or a palmprint. In thefollowing, a biometric imaging device 100 configured to capture an imageof a finger 104 in contact with an outer surface 106 of the displaypanel 102 will be described. The outer surface 106 will thereby bedefined as a sensing surface 106.

The biometric imaging device 100 further comprises a transparentsubstrate 112 arranged to cover the image sensor 108, an opaque layer114 covering an upper surface of the transparent substrate 112. Theopaque layer 114 further comprises a plurality of separate openings 116,arranged at a distance from each other; and a plurality of microlenses118 arranged in an array, such as a hexagonal or regular array. Eachmicrolens 118 is here arranged in a respective opening 116 of the opaquelayer 114 in the same plane as the opaque layer 114. Moreover, themicrolens 118 has the same size and shape as the opening 116 to preventany stray light which has not passed through the microlens 118 fromreaching the image sensor 108.

In another embodiment the microlenses 118 may be arranged above theopaque layer 114, i.e. at a vertical distance from the opaque layer 114,with the focal point of the microlens 118 being located between theopaque layer 114 and the image sensor 108.

Each microlens 118 is configured to redirect light through thetransparent substrate 112 and onto a subarray 120 of pixels in thephotodetector pixel array 109. A subarray 120 is here defined as thearray of pixels which receives light from a corresponding microlens 118.It can be seen that neighboring subarrays 120, 121 overlap, i.e.subarrays resulting from neighboring microlenses 118, 119. The overlapmay be used to determine image reconstruction parameters as will bedescribed in further detail in the following.

It should further be noted that the microlenses 118 and display pixelsare not drawn to scale. The microlenses 118 receives light reflected bythe finger 104 which has propagated through the display panel 102 beforereaching the microlens 118, and the light received by the microlens 118is focused onto the image sensor 108.

FIG. 2 is an exploded view of the biometric imaging device 100 moreclearly illustrating the subarrays 120, 121 of pixels in the pixel array109 receiving light from one microlens 118. The microlenses 118 are hereillustrated as circular plano-convex lenses, providing a circularsubarray 120, 121 of pixels. It can be seen also in FIG. 2 that thesub-images from adjacent microlenses overlap. It would also be possibleto use a rectangular microlens which would lead to an approximatelyrectangular subarray of pixels. The pitch of the microlenses 118 can beconsidered to be known with high accuracy. The pitch may also bedifferent in in X- and Y-directions. All of the microlenses 118 withinthe microlens array are preferably of the same size and shape.

Each microlens 118 thus redirects light onto a pixel array 120comprising a plurality of light sensing elements such that a sub-imageis captured by the subarray 120 for the corresponding microlens 118.Each sub-image represents a portion of the fingerprint. The imageanalysis required to verify a fingerprint after image capture can beperformed in many different ways and will not be discussed in detailherein.

FIG. 3 is a flowchart outlining steps of a method according to variousembodiments of the invention, and the method will be described withfurther reference to the biometric imaging device illustrated in FIGS. 1and 2. The method may be performed by a control unit in the biometricimaging device or by a control unit of a device connected to thebiometric imaging device. The control unit may include a microprocessor,microcontroller, programmable digital signal processor or anotherprogrammable device. The control unit may also, or instead, include anapplication specific integrated circuit, a programmable gate array orprogrammable array logic, a programmable logic device, or a digitalsignal processor. Where the control unit includes a programmable devicesuch as the microprocessor, microcontroller or programmable digitalsignal processor mentioned above, the processor may further includecomputer executable code that controls operation of the programmabledevice.

The method comprises, by the image sensor 108, capturing 300 a pluralityof sub-images 121, 122 together representing an image of a biometricobject 104 in contact with a sensing surface 106 of the biometricimaging device 100, each sub-image corresponding to a respectivemicrolens; and determining 302 a demagnification factor based on atleast a subset of the plurality of sub-images. Capturing an image may bedone by capturing an image using the full area of the image sensor, butit is also possible to capture an image using only a portion of theimage sensor, such as a portion corresponding the location of the fingeron the sensing surface.

Moreover, there are different ways to determine the demagnificationfactor based on the plurality of sub images as will be described in thefollowing.

According to an embodiment outlined by the flowchart in FIG. 4, thebiometric imaging device is configured so that sub-images 121, 122corresponding to adjacent microlenses 118, 119 are partially overlappingas illustrated in FIGS. 1 and 2. The method comprises determining 408 aspatial offset between two captured adjacent sub-images corresponding totwo adjacent microlenses; which can be done by determining a crosscorrelation between the two captured adjacent sub-images. To determinethe cross correlation according to an example embodiment, it is assumedthat the pitch between adjacent microlenses known both in theX-direction and in the y-direction. The microlens pitch is defined inthe manufacturing process and can be determined with a high degree ofaccuracy. In the present context it can be assumed that thedemagnification factor is constant for all microlenses, in practicemeaning that the microlens array, the sensing surface 106 and the imagesensor 108 are substantially planar and parallelly arranged. However,for a device where the mounting of the biometric imaging deviceunderneath e.g. a display panel is tilted in relation to the surface ofthe display panel, the object distance may vary across the image sensorarea. A tilted imaging device means that the demagnification varieslinearly in X and Y directions. It is still possible to estimate thedemagnification(s) of the microlens grid in a situation like this. Inprinciple the demagnification factor can be estimated per each microlenspair and thus in practice per microlens.

First, a subset of microlenses having the same demagnification factor isselected 400. In practice, it is not necessary to use sub-images fromall microlenses in the microlens array when determining the spatialoffset, it is sufficient to use the microlenses which are located belowthe biometric object. Thereby it is sufficient that a subset ofmicrolenses have the same demagnification factor.

Next, an X-dimension and a Y-dimension cross correlation vector for eachpair of microlenses in the subset of microlenses is determined 402, andan average X-dimension and Y-dimension cross correlation vector isformed 404 from the determined cross correlation vectors, followed bysumming 406 the X-dimension and Y-dimension cross correlation vectors.

However, if the microlens pitch in the X-dimension is different from thepitch in the Y-dimension, then the cross-correlation vectors cannot besummed right away. To make summation of X and Y cross correlationvectors possible, one of the vectors is either spatially compressed orspatially expanded so that both vectors match.

It may also be required to resample both the X and the Ycross-correlation vectors before addition. Furthermore, the subimagesmay be spatially filtered before the cross-correlations are calculated.The filters used depends on whether the subimages are used for X or Ydimension correlations. In the present application, spatial band-passfiltering is advantageously used to emphasize the spatial frequenciesexpected to belong to a fingerprint, as the fingerprint object is theuseful signal.

Once the X and Y cross correlation vectors are summed, the spatialoffset can be determined 408 based on the summed cross correlationvector, and the demagnification factor can be determined based on thedetermined spatial offset. In particular, the spatial offset is found asthe maximum index of the correlation vector (fractional value is foundby using an interpolation method). Once the spatial offset is found, thedemagnification is determined 302 by dividing the microlens X pitch withthe spatial offset.

An advantage of the above described method is that it can be usedwithout any prior calibration. This means that the described method canbe used also during the production stage for an initialfactory-calibration, thereby eliminating the steps of performing acalibration using a calibration target. Accordingly, the manufacturingprocess can be simplified which is very important in large scalemanufacturing. Moreover, the method can be automatically be applied whenthe object distance is changed, such as when a screen protector isattached to a smartphone.

FIG. 5 is a flowchart outlining steps of a method according to variousembodiments of the invention. The described method further comprisesperforming 500 stitching to form a full image from a plurality ofsub-images based on a predetermined demagnification factor and a knownmicrolens pitch. Accordingly, in this method a demagnification factor isneeded along with the known microlens pitch. The demagnification factormay for example, be derived by a previous calibration either using thepreviously described method or from a calibration step duringmanufacturing using a calibration image target. It would also bepossible to start from a default demagnification factor which is presetbased on expected properties of the imaging device.

Next, a figure of merit is determined 502 for the full image based on apredetermined set of image properties, such as image contrast and/orfrequency content. The figure of merit is intended to represent thequality of the image with respect to how well the image can be used forsubsequent biometric identification. It should also be noted that thedistinct and repetitive pattern of a biometric feature such as afingerprint simplifies the determination of a figure of merit of theimage since the desirable image properties of a fingerprint are wellknown.

Next, the figure of merit is compared 504 with a predetermined figure ofmerit threshold value. The figure of merit threshold value preferablyrepresents an image quality where biometric identification can beperformed with acceptable accuracy.

If the figure of merit is below the predetermined figure of meritthreshold value, the demagnification factor is changed 506 and a newfigure of merit is determined.

Accordingly, the method may be implemented as brute force method whichcomprises looping over a set of values of the demagnification factor andmeasure a figure of merit in the recombined image for each value of thedemagnification factor followed by choosing the demagnification factorcorresponding to the best figure of merit. The change may be done byiteratively changing the demagnification factor until a figure of meritabove the threshold value is found.

FIG. 6 schematically illustrates a smartphone 600 comprising an opticalbiometric imaging device 102 integrated in the display panel of thesmartphone 100. The optical biometric imaging device 102 is configuredto capture an image of an object 104 in contact with an outer surface106 of the biometric imaging device 102. The object 104 in contact withthe outer surface 106 is here illustrated as a finger 104 in contactwith the surface 106 of the display panel. In addition to fingerprints,the described device 102 may also be used to capture palmprints.

Even though the invention has been described with reference to specificexemplifying embodiments thereof, many different alterations,modifications and the like will become apparent for those skilled in theart. Also, it should be noted that parts of the method may be omitted,interchanged or arranged in various ways, the method yet being able toperform the functionality of the present invention.

Additionally, variations to the disclosed embodiments can be understoodand effected by the skilled person in practicing the claimed invention,from a study of the drawings, the disclosure, and the appended claims.In the claims, the word “comprising” does not exclude other elements orsteps, and the indefinite article “a” or “an” does not exclude aplurality. The mere fact that certain measures are recited in mutuallydifferent dependent claims does not indicate that a combination of thesemeasures cannot be used to advantage.

1. Method for determining image reconstruction parameters in an opticalbiometric imaging device comprising a plurality of microlenses forming amicrolens array and an image sensor arranged to receive light havingpassed through the microlenses, the method comprising: by the imagesensor, capturing a plurality of sub-images together representing animage of a biometric object in contact with a sensing surface of thebiometric imaging device, each sub-image corresponding to a respectivemicrolens, wherein sub-images corresponding to adjacent microlenses arepartially overlapping; and determining a demagnification factor based onan overlap of at least a subset of the plurality of sub-images.
 2. Themethod according to claim 1, wherein the method further comprises:determining a spatial offset between two captured adjacent sub-imagescorresponding to two adjacent microlenses; and determining thedemagnification factor based on the determined spatial offset.
 3. Themethod according to claim 2, wherein determining a spatial offsetcomprises determining a cross correlation between the two capturedadjacent sub-images.
 4. The method according to claim 3, whereindetermining a spatial offset further comprises: selecting a subset ofmicrolenses having the same demagnification factor; determining anX-dimension and an Y-dimension cross correlation vector for each pair ofmicrolenses in the subset of microlenses; forming an average X-dimensionand Y-dimension cross correlation vector from the determined crosscorrelation vectors; summing the X-dimension and Y-dimension crosscorrelation vectors; and determining a spatial offset based on thesummed cross correlation vector.
 5. The method according to claim 4,further comprising determining the demagnification factor based on thespatial offset and a known pitch between adjacent microlenses.
 6. Themethod according to claim 1, further comprising: performing stitching toform a full image from a plurality of sub-images based on apredetermined demagnification factor and a known microlens pitch;determining a figure of merit for the full image based on apredetermined set of image properties; comparing the figure of meritwith a predetermined figure of merit threshold value; and if the figureof merit is below the predetermined figure of merit threshold value,changing the demagnification factor and determining a new figure ofmerit.
 7. The method according to claim 6, wherein the predetermined setof image properties is at least one of image contrast and frequencycontent.
 8. The method according to claim 6, wherein the predetermineddemagnification factor is acquired using a calibration image target. 9.The method according to claim 6, further comprising optimizing thedemagnification factor by iteratively changing the demagnification todetermine a maximum figure of merit.
 10. The method according to claim1, further comprising using the demagnification factor in an imagereconstruction process to form an image to be used for biometricverification.
 11. A biometric imaging device comprising: an imagesensor; and a plurality of microlenses forming a microlens arrayarranged to redirect light from a sensing surface of the imaging devicetowards the image sensor, wherein the image sensor is configured tocapture a plurality of sub-images together representing an image of abiometric object in contact with the sensing surface, wherein sub-imagescorresponding to adjacent microlenses are partially overlapping, eachsub-image corresponding to a respective microlens, and to determine ademagnification factor based on an overlap of at least a subset of theplurality of sub-images.
 12. The biometric imaging device according toclaim 11, wherein the imaging device is further configured to determinea spatial offset between two captured adjacent sub-images correspondingto two adjacent microlenses and determine the demagnification factorbased on the determined spatial offset.
 13. The biometric imaging deviceaccording to claim 11, wherein the microlenses are arranged in the formof a hexagonal array.
 14. An electronic user device comprising abiometric imaging device according to claim 11.